The present invention relates to methods and systems for displaying interactive three-dimensional object movie and panoramas.
The following U.S. Patents have been found in a U.S. Patent Search and are believed to be generally relevant to the field of the invention:
U.S. Pat. No. 4,752,836 June 1988 Blanton et al. . . . 358/342
U.S. Pat. No. 5,396,583 March 1995 Chen et al. . . . 395/127
Interactive three-dimensional object movies and panoramas are becoming increasingly popular in entertainment and advertising over the Internet. These movies and panoramas enable the viewer to interact with the video stream and allow a great deal of functionality, such as the capability to rotate and move objects, zoom in and out, navigate through a panorama, link from one movie to another through the use of hot spots, and much more.
Principles of simulating a panorama from still images on a computer display are described in the following publications:
1. Blanton, K. A., Finlay, W. M., Sinclair, M. J. and Tumblin, J. E., Method and Apparatus for Reproducing Video Images to Simulate Movement within a Multi-Dimensional Space, U.S. Pat. No. 4,752,836, Jun. 21, 1988.
2. Chen, S. E., QuickTime(copyright) VRxe2x80x94An Image Based Approach to Virtual Environment Navigation, SIGGRAPH: Computer Graphics Proceedings, Annual Conference Series, 1995, 29-38.
3. Chen, S. E. and Miller, G. S. P., Cylindrical to Planar Image Mapping using Scanline Coherence, U.S. Pat. No. 5,396,583, Mar. 7, 1995.
Conventional movies use a sequence of frames as input, and advance one frame at a time. Each time a new frame is to be displayed, it is necessary to clear an image buffer of any old frame data, and load the new frame. A view window is used to display part or all of each frame. Rapid advancement through the frames gives the viewer the appearance of continuous motion. In order to transmit a complete movie from a server to a client computer, it is thus necessary for the client to receive all of the frames, and for the frames to be organized into a sequenced database.
In processing large still images extremely high memory requirements may be encountered. An uncompressed 8{fraction (1/2)}xc3x9711 inch photograph scanned in 24-bit color at 300 dpi requires approximately 25 MB of storage. Transmission of large image files is slow, but even after the files are received by the client computer, displaying them and carrying out simple manipulations are computationally intensive and require large amounts of computer memory. In particular, when viewing a sub-sampled version of the image, scaled to fit in a viewer""s monitor screen, the operation of zooming in on a portion of the image can take several minutes on a personal computer containing a PENTIUM(copyright) processor.
Ideally, the viewer would like to focus on specific parts of the image and see various parts of the image at different scales, a process known as interactive xe2x80x9cgazing.xe2x80x9d As zooming progresses, the display shows image tiles of increasingly higher resolution which cover correspondingly smaller areas of the full image. For large images, gazing is an ideal vehicle for information display on a computer monitor, since the image is too large to be displayed in its entirety at full scale. Moreover, whenever a large image is displayed at a low resolution scaled to the size of a computer display screen, the tendency of the viewer is to want to focus in on specific areas at higher resolution; that is, viewers prefer to see portions of the image at higher resolution rather than the full image at low resolutionxe2x80x94portions which they can choose themselves. Conventional techniques make this impractical, however, due to the difficulty and slowness of transmitting and manipulating large digital images.
It is an object of the present invention to enhance the on-line delivery of digital images such as movies and panoramas, by using a novel data structure, which stores pre-manipulated image content in multiple image content units and downloading the data structure in a manner such that any of the stored image content units may be accessed at least a minimum quality prior to completion of downloading.
In accordance with a preferred embodiment of the present invention, the data structure is built by joining all of the image content units together into one large rectangular image. The large image is encoded, transmitted to client computers and decoded on the client computer side.
When the digital image is a movie, each image content unit constitutes a frame and the effect of advancing the frames is accomplished by moving a view window within the large image. The view window being displayed moves in such a way that it maintains the same relative position within each frame within the large image, so that advancement of the view window makes it seem as if the frames were being advanced. This can all be carried out very quickly, faster than if the frames were viewed independently in the conventional manner.
It is another objective of the present invention to enhance the on-line delivery of large still images at multiple resolutions, and to enable fast interactive gazing capability. In large still images, an image content unit may be a tile of the large still image. By applying the invention to multi-resolution tiles making up the image, joined together to form a long horizontal image, the client computer can quickly display a low resolution version of the full image, almost immediately after the data begins to stream, and can zoom in interactively by gazing at areas of interest. The choice as to which areas are of interest is completely in the hands of the producer.
The present invention seeks to simulate an interactive three-dimensional object movie or panorama on a computer display, as data is being received on-line. A multiplicity of individual frames are arranged so as to form a vertical strip. This vertical strip, when entered into the memory as a digital image, can be streamed from a server to a client in horizontal bands, enabling the client to begin viewing the object movie on-line as soon as the first few bands are received, while the server continues to transmit the rest of the data. As more data arrives, the client can view more of the object movie.
In order to view images and video on a computer display, a user, or client, retrieves the media from a storage device or server, typically connected via a network. Bandwidth limitations affect the amount of time required to transmit a video frame from the server to the client, and thus limit the video frame rate. Moreover, when dealing with object movies and panoramas, the images being transmitted are extremely large, so that overcoming bandwidth limitation is a critical enabling factor for the technology, even for high bandwidth networks. Currently, to overcome the limitations, two methods are used: The first is to compress the video frame sequence, thereby speeding up transmission time at the cost of additional processing to decompress the frames prior to display. The second is to copy the entire sequence to an intermediate storage device, such as a hard disk, to which the client has higher bandwidth access, at the cost of delaying the viewing of the video until the entire sequence has been delivered.
It is therefore desirable to provide a method for clients who are linked by a network to a server to view an object movie or panorama on-line. It is also desirable to enable the client to interact with the movie or panorama, by directly controlling the viewing window. A primary advantage of the present invention is to change the layout of the frame sequence from that of typically a long horizontal image in which the frames are joined horizontally left to right, to that of a long vertical image in which the frames are joined vertically top to bottom. The frames are decompressed in sequence as successive horizontal bands are processed. Specifically, the second frame, being positioned underneath the first frame, is only processed after the first frame is complete. Had the frames instead been maintained in the layout of a long horizontal image, each long horizontal band would span all of the frames, and no individual frame in the sequence would be ready for display until all of the frames are decompressed.
The virtual reality effect of the object movie or panorama is obtained by positioning a sliding view window over successive frames, as the user requests a directional move. The position of the display window on the client computer remains fixed, but the window of source image pixel data which is being displayed changes with each view. The availability of whole frame data in succession order enables the object movie or panorama to be progressively displayed. As each additional frame is decompressed, the viewer can move the object or explore the panorama to greater extent. Furthermore, as each additional frame slice is integrated, the quality of the frame is enhanced, until the entire data stream is received by the client. Thus the movie or panorama can begin to be displayed on the client computer almost immediately after transmission of the compressed data from server to client starts.
Had the object movie or panorama frames been compressed individually, as by progressive JPEG, and transmitted in parallel channels, there would be no control over the order in which frames slices are being received, since the individual channel transmissions take place asynchronously. Thus it is seen that joining the frames into a single long image also affords a mechanism for controlling the order in which the compressed frame data arrives at the client side. It is important to emphasize that for systems which require the codec to be compatible with a standard, as is the case with JPEG, for example, the joining of frames into a single long image is fully compliant, and does not necessitate introduction of a non-standard image format for the data streaming. It is also important to emphasize that there is a very large amount of data being transmitted, even after compression, and without progressive rendering the client would have to wait a long time until all of the data arrives and viewing could begin.
The JPEG standard is described by Pennebaker, W. B. and Mitchell, J. L. in JPEG: Still Image Data Compression, Van-Nostrand Reinhold, N.Y., 1993, the disclosure of which is herein incorporated by reference.
According to the present invention there is provided a method for simulating an interactive three-dimensional object movie or panorama on a computer display, as data is received on-line, comprising the steps of preparing a sequence of individual frames from an object movie or panorama video strip to form a long horizontal image, arranging said individual frames in a sequence from top to bottom to form a long vertical image, compressing said long vertical image, storing the compressed data on a server computer storage device, transmitting the compressed data to a client computer, decompressing the compressed data on-line to reconstruct said long vertical image on the client computer, while the compressed data is being transmitted, separating the individual frames from the long vertical image on the client computer, as they become available, re-joining the individual frames into a long horizontal image on the client computer, and progressively displaying sliding window views of said horizontal image to simulate interactive object motion or panorama navigation on the user computer display device, as the data for the complete construction of these views becomes available.
The invention further provides an image processing system operative on an object movie or panorama film strip for providing interactive on-line simulation of a three-dimensional object movie or panorama including an AID converter for converting the film strip into digital data, a production unit for converting the digital data into a long vertical digital image, an encoder for compressing said long vertical image so that it can be efficiently stored on a server computer storage device and further transmitted to a client computer, a transmitter for sending the compressed image data as a data stream to a client computer upon request, and a decoder for decompressing the data stream on the client computer into image data and arranging the image data into a form suitable for interactive display on-line, as the data is received.
The present invention seeks to simulate an interactive movie on a computer display, as data is being received on-line. A sequence of digital frames comprising views as seen within the movie is constructed out of a continuous video or a sequence of photographs taken from different viewing perspectives. The frames are assembled together as tiles to form a single large rectangular shaped image, which is then encoded and stored on a server. For the purposes of the present description, it is assumed specifically that the frames are joined into a single long horizontal image, but this is not essential.
The encoded data is transmitted to client computers and decoded on the client computer side, typically back into a long horizontal image. A viewing window is set up on the client computer side, enabling jumping from one position to another within the long horizontal image, the window remaining in the same position relative to the individual frames, but jumping from frame to consecutive frame. The viewing window is displayed in a fixed location on the client computer monitor. Together, the effect of advancing the viewing window as described above and fixing the display window give an appearance of motion, as if the individual frames of the movie were advancing. xe2x80x9cHot spotsxe2x80x9d can be mapped onto the long horizontal window, so that the viewer can link from one movie to another. When a hot spot is selected, the view window switches from within the current long horizontal image to another such image, corresponding to the movie to be linked, and the viewer then advances within the new movie.
A primary advantage of the present invention is that by embedding a movie within a single long horizontal image, the adjustment of the viewing window can be done by the highly optimized operating system display functions for bitmap images, within the same bitmap. This is faster than changing bitmaps with each successive viewing window, since the image buffer does not have to be continually reloaded. It becomes an even more significant advantage if the long horizontal image is encoded in the JPEG format. Had the movie frames been JPEG encoded independently of one another, without first assembling them together, then to view successive windows it would be necessary to decode successive JPEG images, which would dramatically slow down the playback speed. It would also reduce the data compression factor since (a) there would be more overhead, and (b) entropy encoding, such as Huffman and arithmetic compression, typically performs better when similar data sequences are merged, on account of the larger population on which the statistics base themselves.
Regarding the use of JPEG encoding, an important feature of the present invention is that it is compliant with the JPEG standard, since a single JPEG image is being stored for transmission.
Another advantage of the present invention is that there is no need for the client computer to arrange the frames in a sequenced database, since it is receives only a single image. If the frames had instead been sent independently as individual frames, then they would either have to be sequenced in a single stream, and then unpacked into a database on the client computer side, or else sent in parallel channels, in which case it would be necessary to synchronize the multiple streams. By sending the frames as one single large frame, all of this data management is avoided.
The present invention further includes a production tool for the content provider to carry out the assembling of the frames and the encoding. The tool also includes a decoder, enabling the producer to preview the movie. There is a feedback loop so that the producer can adjust the encoding parameters for fine tuning the production.
The present invention also applies to interactive audio. Rather than represent the audio media as several audio clips stored individually, the producer can join them into one long signal, and move from one segment to another by means of a sliding audio segment. The position of the sliding segment is adjusted in response to user commands interpreted through the mouse and keyboard.
The present invention can be used as an effective means of delivering large still digital images progressively on-line, and to enable fast interactive gazing capability on the part of the viewer. By means of the present invention, the viewer can interactively gaze at the image in areas of interest almost immediately after the streaming is initiated, as soon as a low resolution version is received and displayed, regardless of how large the image is. Upon gazing, the display quickly loads in the requested area of the image at the next higher resolution, enabling a fast zoom-in. To accomplish this, image tiles are formed from the original image at different resolutions and joined together into a long horizontal image.
Hot spots and links are generated to map each area of interest in a tile to the tile containing the area at the next higher resolution. The view window then moves from tiles of one resolution to tiles of the next higher resolution, in response to interactive user mouse clicks on hot spot areas, giving the appearance of zooming in. The choice of which areas of the image to mark with hot spots as areas of interest is in the hands of the producer.
It is also an object of the present invention to provide a method of applying JPEG compression to multiple images wherein the images are joined together end to end in a given direction as one large digital image for JPEG compression and JPEG decompression.
This method is particularly useful for interactively accessing progressive JPEG compressed multiple images prior to completion of progressive JPEG decompression thereof.
There is thus provided in accordance with a preferred embodiment of the present invention a method for presenting a broadcast moving picture including broadcasting stored images including partially overlapping images at different resolutions, and viewing some of the partially overlapping images at different resolutions in a desired sequence thereby to display the moving picture.
Further in accordance with a preferred embodiment of the present invention the stored images being stored with partial redundancy.
There is also provided in accordance with a preferred embodiment of the present invention a method for interactive gazing at a broadcast large digital image including broadcasting partially overlapping stored multiple resolution tiles of the large digital image, and selectably viewing portions of the partially overlapping multiple resolution tiles in a desired sequence thereby to effect interactive gazing.
Further in accordance with a preferred embodiment of the present invention the stored multiple resolution tiles being stored with partial redundancy.
Still further in accordance with a preferred embodiment of the present invention the large digital image being a panorama.
There is also provided in accordance with a preferred embodiment of the present invention apparatus for producing a moving picture including a digital image broadcaster providing individual images including at least partially overlapping images at different resolutions, and a viewer which selectably displays portions of the partially overlapping images at different resolutions in a desired sequence thereby to define the moving picture.
Further in accordance with a preferred embodiment of the present invention the individual images being stored with partial redundancy.
There is also provided in accordance with a preferred embodiment of the present invention apparatus for providing interactive gazing at a broadcast large digital image including a digital image broadcaster providing partially overlapping multiple resolution tiles of the large digital image, and a viewer which may selectably display portions of the partially overlapping multiple resolution tiles in a desired sequence, thereby to effect interactive gazing.
Further in accordance with a preferred embodiment of the present invention the multiple resolution tiles being stored with partial redundancy.
Still further in accordance with a preferred embodiment of the present invention the large digital image being a panorama.
There is also provided in accordance with a preferred embodiment of the present invention a digital image display system including a memory storing digital images including partially overlapping images at different resolutions in an interactively accessible data structure, and a digital data transmitter, downloading and transmitting to a remote user, via a public network, data stored in the memory in a manner such that the partially overlapping images at different resolutions may be interactively accessed by the remote user.
Further in accordance with a preferred embodiment of the present invention the digital images being stored in the memory with partial redundancy.
There is also provided in accordance with a preferred embodiment of the present invention a digital image display method including storing digital images including partially overlapping images at different resolutions in an interactively accessible data structure, and downloading and transmitting to a remote user, via a public network, the digital images in a manner such that the partially overlapping images at different resolution may be interactively accessed by the remote user.
Further in accordance with a preferred embodiment of the present invention the storing using partial redundancy.
There is also provided in accordance with a preferred embodiment of the present invention a method for simulating an interactive movie on a computer display from a sequence of digital frames, as data is received on-line, including the steps of storing partially overlapping image tiles at different resolutions for several of the digital frames, transmitting the partially overlapping image tiles for several of the digital frames to a client computer, and displaying window views of some of the partially overlapping image tiles for some of the digital frames in response to keyboard presses and mouse clicks by a user, to exhibit the same effect as if zooming in and out and advancing forward and backward through the digital frames, in order to simulate interactive motion on the client computer display device.
Further in accordance with a preferred embodiment of the present invention the storing using partial redundancy.
There is also provided in accordance with a preferred embodiment of the present invention an image processing system, operative on a sequence of individual digital frames to produce an on-line interactive movie, including a storage device for archiving partially overlapping image tiles of digital frames at different resolutions, a transmitter for delivering the partially overlapping image tiles of the digital frames at different resolutions to a client computer, and a monitor for displaying window views of some of the partially overlapping image tiles for some of the digital frames in response to keyboard presses and mouse clicks by a user, to exhibit the same effect as if zooming in and out and advancing forward and backward through the digital frames, in order to simulate interactive motion on the client computer display device.
Further in accordance with a preferred embodiment of the present invention the image tiles being stored in the storage device with partial redundancy.
There is also provided in accordance with a preferred embodiment of the present invention a method for progressively displaying a large digital image on a computer display, as data is received on-line, including the steps of forming partially overlapping image tiles at multiple resolutions from the original digital image, storing the image tiles, transmitting some of the image tiles to a client computer, and displaying portions of the image tiles in response to keyboard presses and mouse clicks by a user, exhibiting the same effect as if zooming in and out of the large image, in order to simulate interactive zooming on the client computer display device.
Further in accordance with a preferred embodiment of the present invention the storing using at least partial redundancy.
Still further in accordance with a preferred embodiment of the present invention the large digital image being a panorama.
There is also provided in accordance with a preferred embodiment of the present invention an image processing system for progressively displaying a large digital image on-line, including an image processor for forming partially overlapping image tiles at multiple resolutions from the original large digital image, a storage device for archiving the image tiles, a transmitter for delivering some of the image tiles to a client computer, and a monitor for displaying window views of some of the image tiles, in response to keyboard presses and mouse clicks by a user, to exhibit the same effect as if zooming in and out of the large digital image, in order to simulate interactive zooming on the client computer display device.
Further in accordance with a preferred embodiment of the present invention the image tiles being stored in the storage device with partial redundancy.
Still further in accordance with a preferred embodiment of the present invention the large digital image being a panorama.
There is also provided in accordance with a preferred embodiment of the present invention an image processing system operative on an object movie or panorama film strip for providing interactive on-line simulation of a three-dimensional object movie or panorama including an A/D converter for converting the film strip into digital data, a production unit for using the digital data to produce partially overlapping images at different resolutions, a storage device for storing the partially overlapping images at different resolutions, and a transmitter for sending some of the partially overlapping images at different resolutions to a client computer for interactive display on-line.
Further in accordance with a preferred embodiment of the present invention the partially overlapping images being stored in the storage device with partial redundancy.
There is also provided in accordance with a preferred embodiment of the present invention a method for producing interactive media including providing partially overlapping individual interactively accessible digital content units, defining a pointer which may be selectably pointed at one of the individual interactively accessible digital content units, and using the pointer to access portions of the individual interactively accessible digital content units in a desired sequence.
There is also provided in accordance with a preferred embodiment of the present invention apparatus for producing interactive media, including a digital file generator providing partially overlapping interactively accessible digital content units, a pointer which may be selectably pointed to one of the partially overlapping interactively accessible digital content units, and a user-operated pointer controller for using the pointer to access portions of the interactively accessible digital content units in a desired sequence.
There is also provided in accordance with a preferred embodiment of the present invention a method for producing a moving picture including providing a digital image including the content of several individual images, defining a view window which may be selectably superimposed over at least a portion of the digital image at a given time, and moving the view window over the digital image such as to view portions of the individual images in a desired sequence thereby defining the moving picture.
Further in accordance with a preferred embodiment of the present invention the individual images comprise images of an object in different orientations, whereby moving the view window over the individual images in different orientations provides a sensation of viewing movement of the object.
Still further in accordance with a preferred embodiment of the present invention the view window is superimposed sequentially over a portion of each of the individual images, so as view the same portion of the object in each of the individual images.
There is also provided in accordance with a preferred embodiment of the present invention a method for gazing at a large still image including providing a digital image including the content of partially overlapping multiple resolution tiles of the large still image, defining a view window which may be selectably superimposed over a portion of the digital image at a given time, and moving the view window over the digital image such as to view portions of the multiple resolution tiles in a desired sequence thereby to effect interactive gazing.
There is also provided in accordance with a preferred embodiment of the present invention apparatus for producing a moving picture including a digital image generator providing a digital image including the content of individual images, a view window which may be selectably superimposed over at least a portion of the digital image at a given time, and a user-operated view window controller for moving the view window over the digital image such as to view at least portions of the individual images in a desired sequence thereby to define the moving picture.
Further in accordance with a preferred embodiment of the present invention the individual images comprising individual images of an object in different orientations, whereby moving the view window over the individual images in different orientations provides a sensation of viewing movement of the object.
Still further in accordance with a preferred embodiment of the present invention the view window being superimposed sequentially over a portion of each of the individual images, so as to view the same portion of the object in each of the individual images.
There is also provided in accordance with a preferred embodiment of the present invention apparatus for providing gazing at a large still image including a digital image generator providing a digital image including the content of partially overlapping multiple resolution tiles of the large still image, a view window which may be selectably superimposed over at least a portion of the digital image at a given time, and a user-operated view window controller moving the view window over the digital image such as to view portions of the multiple resolution tiles in a desired sequence, thereby to effect interactive gazing.
There is also provided in accordance with a preferred embodiment of the present invention a digital image display system including a memory storing digital images in an interactively accessible data structure, which stores pre-manipulated image content in image content units, and a downloader downloading data stored in the memory in a manner such that any of the stored image content units may be accessed at at least a minimum quality prior to completion of downloading.
Further in accordance with a preferred embodiment of the present invention the data structure joins all of the image content units together into one large digital image.
Still further in accordance with a preferred embodiment of the present invention an encoder for encoding the large digital image and wherein the downloader transmits the encoded large digital image to a client computer where it is decoded.
Additionally in accordance with a preferred embodiment of the present invention the large digital image comprising a movie and each image content unit constituting a frame of the movie.
Moreover in accordance with a preferred embodiment of the present invention the large digital image comprising a large still image and each image content unit constituting a tile of the large still image at a given resolution, the image content units being redundant tiles of the large still image at varying resolutions.
Further in accordance with a preferred embodiment of the present invention a user-operated controller for advancing frames in the movie by moving a view window within the large digital image.
Still further in accordance with a preferred embodiment of the present invention the view window moves in such a way that it maintains the same relative position within each frame within the movies, so that advancement of the view window makes it seem as if the frames are being advanced.
Additionally in accordance with a preferred embodiment of the present invention a user-operated controller for user selectable gazing in the large still image by moving a view window within the large digital image.
There is also provided in accordance with a preferred embodiment of the present invention a digital image display method including storing digital images in an interactively accessible data structure, which stores pre-manipulated image content in image content units, and downloading data stored in the memory in a manner such that any of the stored image content units may be accessed at at least a minimum quality prior to completion of downloading.
Further in accordance with a preferred embodiment of the present invention the data structure joins all of the image content units together into one large digital image.
Still further in accordance with a preferred embodiment of the present invention encoding the large digital image and wherein the downloading transmits the encoded large digital image to a client computer where it is decoded.
Additionally in accordance with a preferred embodiment of the present invention the large digital image comprises a movie and each image content unit constitutes a frame of the movie.
Moreover in accordance with a preferred embodiment of the present invention the large digital image comprises a large still image and each image content unit constitutes a tile of the large still image at a given resolution, the image content units comprising redundant tiles of the large still image at varying resolutions.
Further in accordance with a preferred embodiment of the present invention advancing frames in the movie by moving a view window within the large digital image.
Still further in accordance with a preferred embodiment of the present invention the view window moves in such a way that it maintains the same relative position within each frame within the movies, so that advancement of the view window makes it seem as if the frames are being advanced.
Additionally in accordance with a preferred embodiment of the present invention user selectable gazing in the large still image by moving a view window within the large digital image.
Moreover in accordance with a preferred embodiment of the present invention the encoding and decoding operating according to a progressive JPEG algorithm.
There is also provided in accordance with a preferred embodiment of the present invention a method for simulating an interactive movie on a computer display from a sequence of digital frames, as data is received on-line, including joining the digital frames together to form a large digital image encoding the large digital image to generate encoded image data storing the encoded image data transmitting the encoded image data to a client computer upon request decoding the encoded image data to generate a reconstructed large digital image, and displaying moving window views of the reconstructed large digital image in a fixed display window, in response to keyboard presses and mouse clicks by a user, to exhibit the same effect as if advancing forward and backward through the digital frames, in order to simulate interactive motion on the client computer display device.
Further in accordance with a preferred embodiment of the present invention the steps of joining, encoding, storing, transmitting, and decoding are applied to several digital frame sequences, corresponding to several movies, linked together through hot spots.
Still further in accordance with a preferred embodiment of the present invention the step of storing includes storing hot spot locations and link information, along with the encoded image data.
Additionally in accordance with a preferred embodiment of the present invention transmitting hot spot locations and link information along with the encoded image data.
Moreover in accordance with a preferred embodiment of the present invention the step of displaying also uses said hot spot locations, and further includes the steps of processing mouse clicks by the user in hot spot locations to link to another of the reconstructed large digital images, and moving a window view from a position within the current large digital image to a position within the large digital image linked thereto at a hot spot.
Additionally in accordance with a preferred embodiment of the present invention a portion of the large digital image linked thereto is stored within the encoded data for the large digital image linked therefrom, and wherein moving the window view includes moving a window view from a position within the current large digital image linked therefrom to another position within the large digital image linked therefrom corresponding to the location of the portion of the image linked thereto contained therein, and moving a window view from the position within the large digital image linked therefrom to a corresponding position within the large digital image linked thereto.
Further in accordance with a preferred embodiment of the present invention the large digital image being a long horizontal image containing all of the individual movie frames .
Moreover in accordance with a preferred embodiment of the present invention the encoding and decoding operating according to a progressive JPEG algorithm.
There is also provided in accordance with a preferred embodiment of the present invention an image processing system, operative on a sequence of individual digital frames to produce an on-line interactive movie, including a stitching unit for assembling the digital frames together to form a large digital image an encoder for compressing the large digital image into encoded image data a storage device for archiving the encoded image data a transmitter for delivering the encoded image data to a client computer upon request a decoder for decompressing the encoded image data, to generate a reconstructed large digital image, and a monitor for displaying moving window views of the reconstructed large digital image in a fixed display window, in response to keyboard presses and mouse clicks by a user, exhibiting the same effect as if advancing forward and backward through the digital frames, in order to simulate interactive motion on the client computer display device.
Further in accordance with a preferred embodiment of the present invention the stitching unit, encoder, storage device, transmitter and decoder operating on several digital frame sequences, corresponding to several movies, linked together through hot spots.
Still further in accordance with a preferred embodiment of the present invention the storage device archiving the location and link information of the hot spots along with the encoded image data.
Additionally in accordance with a preferred embodiment of the present invention the transmitter delivers the hot spot and link information, along with the encoded image data.
Moreover in accordance with a preferred embodiment of the present invention the monitor using hot spot locations to display moving window views, by means of a processing unit for processing mouse clicks by the user in hot spot locations in order to link to another of the reconstructed large digital images, and a pointer in memory for moving the window view from a position within the current large digital image to a position within the large digital image which is linked thereto at the hot spot locations.
Moreover in accordance with a preferred embodiment of the present invention a portion of the large digital image linked thereto is stored within the encoded data for the large digital image linked therefrom, and wherein the pointer in memory includes a pointer in memory for moving the window view from a position within the current large digital image linked therefrom to another position within the large digital image linked therefrom corresponding to the location of the portion of the image linked thereto contained therein, and another pointer in memory for moving the window view from the position within the large digital image linked therefrom to a corresponding position within the large digital image linked thereto.
Further in accordance with a preferred embodiment of the present invention the large digital image being a long horizontal image containing individual movie frames.
Moreover in accordance with a preferred embodiment of the present invention the encoder and decoder operating according to a progressive JPEG algorithm.
There is also provided in accordance with a preferred embodiment of the present invention a method for simulating interactive audio on a computer sound card from a sequence of digital audio clips, as data is received on-line, including joining the digital clips together to form a long digital signal encoding the long signal to generate encoded audio data, storing the encoded audio data, transmitting the encoded audio data to a client computer upon request, decoding the encoded audio data to generate a reconstructed long digital signal, and playing changing segments of the reconstructed long signal on a sound card, in response to keyboard presses and mouse clicks by a user, to exhibit the same effect as if advancing through the digital audio clips.
There is also provided in accordance with a preferred embodiment of the present invention an audio processing system operative on a sequence of digital audio clips, to produce on-line interactive audio, including a stitching unit for joining the digital audio clips together to form a long digital signal, an encoder for compressing the long digital signal into encoded audio data, a storage device for archiving the encoded audio data, a transmitter for delivering the encoded audio data to a client computer upon request, a decoder for decompressing the encoded audio data to generate a reconstructed long digital signal, and a sound player for playing changing segments of the reconstructed signal, in response to keyboard presses and mouse clicks by a user, exhibiting the same effect as if advancing through the digital audio clips.
There is also provided in accordance with a preferred embodiment of the present invention a method for progressively displaying a large digital image on a computer display, as data is received on-line, including forming image tiles from the original digital image at multiple resolutions, joining the image tiles together to form a large digital image, encoding the large digital image to generate encoded image data, storing the encoded image data, transmitting the encoded image data to a client computer upon request, decoding the encoded image data to generate a reconstructed large digital image, and displaying moving window views of the reconstructed horizontal image in a fixed display window, in response to keyboard presses and mouse clicks by a user, exhibiting the same effect as if zooming in and out of the large image, in order to simulate interactive zooming on the client computer display device.
Further in accordance with a preferred embodiment of the present invention the large digital image being a long horizontal image containing all of the image tiles.
Moreover in accordance with a preferred embodiment of the present invention the encoding and decoding operating according to a progressive JPEG algorithm.
There is also provided in accordance with a preferred embodiment of the present invention an image processing system for progressively displaying a large still image on-line, including an image processor for forming image tiles from the original large still image at multiple resolutions, a stitching unit for joining the image tiles together to form a large digital image, an encoder for compressing the large digital image into encoded image data, a storage device for archiving the encoded image data, a transmitter for delivering the encoded image data to a client computer upon request, a decoder for decompressing the encoded image data to generate a reconstructed large digital image, and a monitor for displaying moving window views of the reconstructed large digital image in a fixed display window, in response to keyboard presses and mouse clicks by a user, to exhibit the same effect as if zooming in and out of the large still image, in order to simulate interactive zooming on the client computer display device.
Further in accordance with a preferred embodiment of the present invention the large digital image being a long horizontal image containing all of the image tiles.
Still further in accordance with a preferred embodiment of the present invention the encoder and decoder operating according to a progressive JPEG algorithm.
There is also provided in accordance with a preferred embodiment of the present invention a method for simulating an interactive three-dimensional object movie or panorama on a computer display, as data is received on-line, including arranging a sequence of individual digital frames from an object movie or panorama video strip in a sequence from top to bottom to form a long vertical image, compressing the long vertical image, storing the compressed data, transmitting the compressed data to a client computer decompressing the compressed data on-line to reconstruct the long vertical image on the client computer, while the compressed data is being transmitted, and progressively displaying selected window views to simulate interactive object motion or panorama navigation on the client computer display device, as the data for the complete construction of these views becomes available.
Further in accordance with a preferred embodiment of the present invention separating the individual frames from the long vertical image on the client computer, as they become available, and joining the individual frames into a large digital image on the client computer, and wherein the step of progressively displaying operating by displaying sliding window views of the large digital image.
Still further in accordance with a preferred embodiment of the present invention the compressing of the long vertical image is performed by encoding horizontal bands of image data in succession.
Additionally in accordance with a preferred embodiment of the present invention the compressing is performed by a JPEG encoder.
Moreover in accordance with a preferred embodiment of the present invention the compressing of the long vertical image is carried out in a progressive mode, resulting in a series of compressed data slices.
Further in accordance with a preferred embodiment of the present invention the compressing is performed by a progressive mode JPEG encoder.
Still further in accordance with a preferred embodiment of the present invention the storing of the compressed data creates a single stream database of compressed image data on the server, and the transmission of the compressed data uses a single data channel.
Additionally in accordance with a preferred embodiment of the present invention the storing of the compressed data creates a multiple stream database of compressed image data on the server, and the transmission of the compressed data uses multiple data channels.
Moreover in accordance with a preferred embodiment of the present invention the decompressing of the compressed data is performed by decoding horizontal bands of image data in succession.
Further in accordance with a preferred embodiment of the present invention the decompressing is performed by a JPEG decoder.
Still further in accordance with a preferred embodiment of the present invention the decompressing of the compressed data is carried out in a progressive mode, resulting in a series of decompressed data slices.
Additionally in accordance with a preferred embodiment of the present invention the decompressing is performed by a progressive mode JPEG decoder.
Moreover in accordance with a preferred embodiment of the present invention the decompressing of the compressed data comprises the steps of decompressing the individual compressed image slices, and integrating the slices as they become available so as to obtain successively enhanced video frames.
Further in accordance with a preferred embodiment of the present invention the progressive displaying is carried out by a viewing window which jumps from frame to frame, as each successive frame is made available.
Still further in accordance with a preferred embodiment of the present invention the progressive displaying is carried out by a viewing window which moves smoothly from frame to frame, as each successive frame is made available.
There is also provided in accordance with a preferred embodiment of the present invention apparatus for simulating an interactive three-dimensional object movie or panorama on a computer display, as data is received on-line, including a frame composer arranging a sequence of individual digital frames from an object movie or panorama video strip in a sequence from top to bottom to form a long vertical image, a compressor compressing the long vertical image, a memory storing the compressed data, a transmitter transmitting the compressed data to a client computer, a decompressor decompressing the compressed data on-line to reconstruct the long vertical image on the client computer, while the compressed data is being transmitted, and a display driver progressively displaying selected window views to simulate interactive object motion or panorama navigation on the client computer display device, as the data for the complete construction of these views becomes available.
Further in accordance with a preferred embodiment of the present invention a frame separator separating the individual frames from the long vertical image on the client computer, as they become available, and a stitching device joining the individual frames into a large digital image on the client computer, wherein the display driver operates by displaying sliding window views of the large digital image.
Still further in accordance with a preferred embodiment of the present invention the compressor operating on the long vertical image by encoding horizontal bands of image data in succession.
Additionally in accordance with a preferred embodiment of the present invention the compressor comprising a JPEG encoder.
Moreover in accordance with a preferred embodiment of the present invention the compressor operates on the long vertical image in a progressive mode, resulting in a series of compressed data slices.
Further in accordance with a preferred embodiment of the present invention the compressor comprises a progressive mode JPEG encoder.
Still further in accordance with a preferred embodiment of the present invention the memory including a single stream database of compressed image data on the server, and the transmitter employing a single data channel.
Additionally in accordance with a preferred embodiment of the present invention the memory including a multiple stream database of compressed image data on the server, and the transmitter employing multiple data channels.
Moreover in accordance with a preferred embodiment of the present invention the decompressor operating by decoding horizontal bands of image data in succession.
Further in accordance with a preferred embodiment of the present invention the decompressor comprising a JPEG decoder.
Still further in accordance with a preferred embodiment of the present invention the decompressor operating on the compressed data in a progressive mode, resulting in a series of decompressed data slices.
Additionally in accordance with a preferred embodiment of the present invention the decompressor comprising a progressive mode JPEG decoder.
Moreover in accordance with a preferred embodiment of the present invention the decompressor operating on the compressed data by decompressing the individual compressed image slices, and integrating the slices as they become available so as to obtain successively enhanced video frames.
Further in accordance with a preferred embodiment of the present invention the display driver operating by causing a viewing window to jump from frame to frame, as each successive frame is made available.
Still further in accordance with a preferred embodiment of the present invention the display driver operates by causing a viewing window to move smoothly from frame to frame, as each successive frame is made available.
There is also provided in accordance with a preferred embodiment of the present invention an image processing system operative on an object movie or panorama film strip for providing interactive on-line simulation of a three-dimensional object movie or panorama including an A/D converter for converting the film strip into digital data, a production unit for converting the digital data into a long vertical digital image, an encoder for compressing the long vertical image so that it can be efficiently stored on a server computer storage device and further transmitted to client computers, a transmitter for sending the compressed image data as a data stream to a client computer upon request, and a decoder for decompressing the data stream on the client computer into image data, and arranging the image data into a form suitable for interactive display on-line, as the data is received.
Further in accordance with a preferred embodiment of the present invention the encoder comprising a progressive compressor and a bit rate controller, for producing compressed image slices.
Still further in accordance with a preferred embodiment of the present invention the decoder comprises a progressive decompressor for the compressed slices and integrator for accumulating the slices into image data.
There is also provided in accordance with a preferred embodiment of the present invention a method for producing interactive media including providing a digital file including content of individual interactively accessible digital content units, defining a pointer which may be selectably pointed at at least a portion of the digital file at a given time, and moving the pointer over the digital file so as to access portions of the individual interactively accessible digital content units in a desired sequence.
Further in accordance with a preferred embodiment of the present invention storing the digital file in an interactively accessible data structure, and downloading the digital file in a manner such that any of the stored multiplicity of individual interactively accessible digital content units may be accessed at at least a minimum quality prior to completion of downloading.
There is also provided in accordance with a preferred embodiment of the present invention apparatus for producing interactive media, said apparatus including a digital file generator providing a digital file including the content of interactively accessible digital content units, a pointer which may be selectably pointed to a portion of the digital file at a given time, and a user-operated pointer controller for moving the pointer over the digital image such as to view portions of the interactively accessible digital content units in a desired sequence.
Further in accordance with a preferred embodiment of the present invention a memory storing the digital file in an interactively accessible data structure, and a downloader downloading data stored in the memory in a manner such that any of the stored individual interactively accessible digital content units may be accessed at at least a minimum quality prior to completion of downloading.
The following definitions are employed throughout the specification and claims:
RESOLUTIONxe2x80x94The relationship between the number of digital samples per unit of an original and the number of digital samples per unit in a rendered version thereof. Specifically, when dealing with images, resolution refers to the relationship between the number of pixels per unit area of an original image or scene and the number of pixels per unit area in a displayed image.
QUALITYxe2x80x94The degree to which a rendered version of an original is faithful to the original. Specifically, when dealing with images, quality refers to the degree to which the displayed image is faithful to the original image or scene. Normally this is expressed as the degree to which the approximation of pixel values in the displayed image approaches the correct pixel values in the original image or scene. Specifically, when dealing with audio, quality refers to the degree to which a played sound is faithful to the original sound.
FRAMExe2x80x94A portion of an original which can be independently and interactively manipulated. Specifically, when dealing with images, frame refers to a portion of an image or of a collection of images which can be independently and interactively manipulated.
TILExe2x80x94A window sized pixel array of a predetermined given size forming part of an image. For example, tiles partition an image into a plurality of arrays, each of which contains an identical number of pixels.
BANDxe2x80x94A collection of digital image pixels which form a horizontal strip of the image. Typically image compression and decompression proceeds in-bands. Within the JPEG algorithm, for example, the bands are 8 pixels wide.
SLICExe2x80x94A sequentially transmitted collection of partial data which is used to build multiple frames. The frames are built up of one or more sequentially transmitted slices, whose contents are accumulated. Specifically, when dealing with images, the slice contains image data. Specifically when dealing with audio, the slice contains audio data.
INTERACTIVExe2x80x94characterizing an application or functionality in which user selected portions of data in a file are repeatedly accessed in a typical operation.